System and method for discounting of historical click through data for multiple versions of an advertisement

ABSTRACT

The present invention is directed towards systems and methods for determining the performance of a plurality of versions of a given advertisement. The method of the present invention comprises retrieving a first version of an advertisement and associated click through data, and retrieving a second version of the advertisement and associated click through data. A clickability score is calculated for the first version of the advertisement using the click through data associated with the first version, and a clickability score is calculated for the second version of the advertisement using the click through data associated with the second version. A difference in clickability scores is determined between the first and second advertisement. The clickability score associated with the first version of the advertisement is modified based upon the difference in clickability scores.

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosures, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE INVENTION

The present invention generally provides methods and systems forfacilitating the ranking and positioning of advertisements. Morespecifically, the present invention provides methods and systems thatuse combinations of analytics data, confidence scores, and expectedrevenue to determine an ordering or positioning of one or moreadvertisements in response to a given request received from a clientdevice.

Advertisements are commonly used on the Internet to promote variousproducts and services. Advertisements may comprise banner ads, links toweb pages, images, video, text, etc. The various advertisements used topromote products on the Internet may be displayed according to a varietyof formats, such as within a ranked result set in response to a searchquery, embedded in a web page, a pop-up, etc. The advertisementsdisplayed to a user of a client device may be selected, redirecting auser to the advertiser's website providing information regarding theproduct or service advertised.

Client devices, communicatively coupled to a network such as theInternet, are able to access various websites that may displayadvertisements. For example, a user of a client device may submit asearch query comprising one or more terms to a search engine, whichcauses the search engine to retrieve a result set comprising links tocontent and advertisements responsive to the search terms provided by auser. The search engine displays the result set to a user, who may thenselect or view items in the result set comprised of one or moreadvertisements.

While the Internet provides advertisers with the ability to reach asignificant quantity of users, thereby increasing the likelihood that agiven product or service is purchased by a user, Internet advertisingfurther provides revenue to search engines that retrieve and displayadvertisements in response to various requests. A search engine maygenerate profit when a given advertisement in a result set is selectedby a user or displayed to a user in response to a search query. Forexample, advertisers may pay a predetermined amount to have banners,images, or other advertisements displayed to users in response to agiven search query (“impressions”). Search engines may also chargeadvertisers for each advertisement selected by a user or delivered to auser (“click through”). However, different advertisements may be more orless profitable for a search engine to display as the amount paid byeach advertiser for displaying or having a user select an advertisementvaries. Furthermore, an advertisement ranked and displayed first in aresult set is more likely to be selected by a given user than anadvertisement ranked second, third, etc. Therefore, advertisers oftenwant their one or more advertisements displayed in positions thatincrease the likelihood that a given advertisement is selected.

A bidding marketplace may be used whereby advertisers may bid for theposition or rank of a given advertisement in response to a given query.Current methods for selecting the position or rank of an advertisementusing a bidding marketplace may select an advertisement with thegreatest bid to appear first in a ranked list of advertisements or in aposition most likely to be seen and selected by users. Such methods forselecting advertisements with the greatest bid are based upon theassumption that advertisements with the greatest bids will yield thegreatest revenues for a search engine. However, while an advertisementwith the greatest bid may generate the most profit when selected, anadvertisement with a lower bid may generate more profit if thelikelihood of users selecting the second advertisement is significantlygreater.

In addition to ranking methods based upon bids associated withadvertisements, result sets comprised of advertisements generated inresponse to a user's query may also be ranked according to an algorithmthat determines the frequency with which a query term appears in a givenweb site. Such ordering methodologies, however, are tailored to provideusers with results that are ordered according to their relevancy withrespect to a given query. Such ordering methods order results, such asadvertisements, in descending order with the result most closely relatedto a user's query ranked first. Ordering advertisements in a result setin descending order with respect to relevancy is based upon theassumption that an advertisement ranked first in a result set displayedto a user is more likely to be viewed or selected than an advertisementranked second, third, etc. However, these methods fail to take intoaccount the profit interests and needs of the search engine providingthe result on behalf of an advertiser.

While the ranking of an advertisement in a ranked result set is oftenindicative of the likelihood that a given advertisement is selected, theadvertisement's appeal to users of client devices may play an importantrole in generating revenue for a search engine. Advertisers may pay asearch engine for each user that selects a given advertisement.Advertisements that are the most appealing and most relevant to a givenuser's query may often be the advertisements most frequently selected byusers, thereby generating greater revenue for a search engine. As notedabove, current methodologies for ordering advertisements in response toa request examine the relevancy of a given advertisement. Whilerelevancy is often a factor in a user's selection of a givenadvertisement, the likelihood of a user selecting a given advertisementis also based upon the advertisement's appeal to users, which may bedetermined based upon previous users' interactions with theadvertisement. Therefore, when determining an ordering foradvertisements in response to a request, search engines may want toconsider relevancy, appeal and revenue to increase the likelihood thatthe most relevant and profitable advertisements are displayed andselected by users.

Current methods and systems for ranking advertisements in a result setor positioning items in a given web page fail to take into accountrelevance, appeal and revenue generating potential of advertisementsdisplayed in response to a search request. In order to overcomeshortcomings associated with existing techniques for ranking orpositioning one or more advertisements, embodiments of the presentinvention provide systems and methods for examining the relevancy,appeal, and revenue generating potential of one or more advertisementsand ranking or positioning the one or more advertisements based upon thesame.

SUMMARY OF THE INVENTION

The present invention is directed towards systems and methods fordetermining the performance of a plurality of versions of a givenadvertisement. According to one embodiment of the invention, a firstversion of an advertisement and click through data associated with thefirst version of the advertisement are retrieved. A second version ofthe advertisement and click through data associated with the secondversion of the advertisement are retrieved. A clickability score iscalculated for both the first and second version of the advertisementusing the click through data for each respective version. Theclickability score associated with each version of the advertisement isused to measure the difference between the first version of theadvertisement and the second version of the advertisement. Theclickability score associated with the first version of theadvertisement is modified based upon the calculated difference betweenthe clickability score associated with the version of the advertisementand the clickability associated with the second version of theadvertisement.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is illustrated in the figures of the accompanying drawingswhich are meant to be exemplary and not limiting, in which likereferences are intended to refer to like or corresponding parts, and inwhich:

FIG. 1. is a block diagram presenting a system for retrieving andranking one or more advertisements based upon a request received from aclient device.

FIG. 2 is a flow diagram presenting a method for retrieving and orderingone or more advertisements based upon expected revenue in response to arequest and delivering the one or more advertisements to client devices.

FIG. 3 is a flow diagram presenting a method for calculating the pricecharged to an advertiser in a bidding marketplace.

FIG. 4 is a flow diagram presenting a method for selecting thepositioning of one or more advertisements in a given web page.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description of the preferred embodiment, reference ismade to the accompanying drawings that form a part hereof, and in whichis shown by way of illustration a specific embodiment in which theinvention may be practiced. It is to be understood that otherembodiments may be utilized and structural changes may be made withoutdeparting from the scope of the present invention.

FIG. 1 presents a block diagram illustrating one embodiment of a systemfor retrieving and ordering advertisements. According to the system ofFIG. 1, a search provider 100 comprises one or more software andhardware components to facilitate retrieving and ordering advertisementsincluding, but not limited to, an index component 102, an advertisementdata store 104, a scoring component 106, a search engine 112, aranking/positioning component 110 and an analytics data store 108. Thesearch provider 100 is communicatively coupled to a network 116, whichmay include a connection to one or more local and/or wide area networks,such as the Internet.

According to one embodiment of the invention, a search engine 112receives search requests from client devices 124 a, 124 b and 124 ccommunicatively coupled to the network 116. A client device 124 a, 124 band 124 c may be any device that allows for the transmission of searchrequests to the search provider 100, as well as receipt of a rankedresult set of advertisements from the search provider 100. According toone embodiment of the invention, a client device 124 a, 124 b and 124 cis a general purpose personal computer comprising a processor, transientand persistent storage devices, input/output subsystem and bus toprovide a communications path between components comprising the generalpurpose personal computer. For example, a 3.5 GHz Pentium 4 personalcomputer with 512 MB of RAM, 40 GB of hard drive storage space and anEthernet interface to a network. Other client devices are considered tofall within the scope of the present invention including, but notlimited to, hand held devices, set top terminals, mobile handsets, etc.The client device 124 a, 124 b and 124 c typically runs softwareapplications such as a web browser that provide for transmission ofsearch requests, as well as receipt and display of ranked result setswhich may comprise of one or more advertisements.

The search engine 112 receives a search request from a given clientdevice 124 a, 124 b and 124 c, and investigates the index 102 toidentify one or more advertisements responsive to the search request.The search engine 112 retrieves one or more advertisements from theadvertisement data store 104 on the basis of the investigation of theindex 102. The one or more advertisements retrieved from theadvertisement data store 104 may be retrieved based on one or more matchtypes. For example, advertisements constituting exact matches may beretrieved, where an exact match may comprise an advertisement whereinthe advertisement contains all of the terms of a given search queryreceived from a client device 124 a, 124 b and 124 c. Similarly,advertisements constituting a broad match may be retrieved, where abroad match comprises an advertisement wherein at least one of thesearch terms of a given query appear in the advertisement. According toanother embodiment of the invention, one or more categories ofadvertisements may be retrieved in response to the request. For example,a category may comprise one or more advertisements related to a givenproduct or service.

The search engine 112 generates a result set that comprises one or moreadvertisements or links to advertisements that fall within the scope ofthe search request. Advertisements may comprise banner ads, text, audiofiles, video files, etc., selectable by a user of a client device 124 a,124 b and 124 c. Advertisements may also comprise links toadvertisements that may be selected by a given user of a client device124 a, 124 b and 124 c.

The result set comprising the advertisements retrieved from theadvertisement data store 104 is delivered to the scoring component 106.The scoring component 106 may calculate a relevance score for eachadvertisement comprising the result set. According to one embodiment ofthe invention, a relevance score comprises an indication of therelatedness of the one or more terms comprising an advertisement and theone or more terms comprising a given search query. For example, anadvertisement retrieved from the advertisement data store may comprisecontent as well as one or more items identifying the advertisement, suchas title, topic, category, etc. A relevance score may calculate howclosely a given advertisement's title, topic, category, etc., as well ascontent, relate to a given query. A relevance score may be calculatedfor the one or more advertisements comprising the result set.

The one or more items comprising the result set may be given anadditional relevance confidence score that may indicate the reliabilityof the calculated relevance score. More specifically, the relevanceconfidence score provides an indication of the extent to which a givenadvertisement matches a given query and the reliability of the givenadvertisement. According to one embodiment of the invention, relevanceconfidence scores are based upon the advertiser providing a givenadvertisement. For example, the relevance confidence score associatedwith an advertisement received from The U.S. Navy constituting an exactmatch with a given query may be increased, as the advertiser (The Navy)is a well-known source. Alternatively, the relevance confidence scoreassociated with an advertisement for a small business that is not wellknown may be decreased, even though the advertisement may be an exactmatch with a given query. According to another embodiment of theinvention, relevance confidence scores may be based upon the type ofmatch. For example, the relevance confidence score associated with anadvertisement constituting an exact match may be increased, while therelevance confidence score associated with a broad match may bedecreased. According to another embodiment of the invention, relevanceconfidence scores may be based upon an examination of an indexidentifying unreliable sources. For example, an index may be maintainedidentifying URLs that direct users to questionable or unreliablecontent, thereby resulting in a decreased relevance confidence score.Relevance confidence scores, according to one embodiment of theinvention, are based upon a scale from zero to one.

Click through data regarding served advertisements is retrieved andstored in the analytics data store 108, which is communicatively coupledto the network 116. The analytics data store 108 is operative to receivedata indicating that a given advertisement was selected by a user of aclient device 124 a, 124 b and 124 c. The analytics data store 108 mayfurther maintain data indicating the time a given advertisement wasselected, user characteristics of a user who selected a givenadvertisement, as well as the position of an advertisement whendisplayed and selected by a user of a client device 124 a, 124 b and 124c.

The scoring component 106 is further operative to use the click throughdata stored in the analytics data store 108 to calculate a normalizedestimate of the click through rate of the one or more advertisementscomprising the result set. A normalized click through rate provides anindication of the likelihood of a user selecting a given advertisementbased upon the past behavior of one or more users with respect to thegiven advertisement. The analytics data store, which maintains clickthrough data for the one or more advertisements comprising the resultset, provides click through data to the scoring component 106. Thescoring component 106 uses the data received from the analytics datastore 108 to calculate a normalized click through rate for the one ormore advertisements comprising the result set.

According to one embodiment of the invention, a normalized click throughrate is calculated to normalize differences in click through rates dueto advertisements appearing in different positions within a rankedresult set. Because advertisements appearing first in a ranked resultset may be more likely to be selected by a user than advertisementsappearing second, third, etc., an advertisement appearing first in aranked result set may have a significantly greater click through ratethan an advertisement appearing second, third, etc. A normalization ofthe click through rates may be performed to compensate for thedifference in rank positions of the one or more items comprising a groupof advertisements. Reference data may be used by the respectivenormalizing algorithm to provide information specifying the likelihoodof an advertisement being selected by a user when displayed in positionone, two, three, etc., of a ranked group of advertisements.

According to another embodiment of the invention, a normalized clickthrough rate is calculated to normalize differences in click throughrates due to advertisements appearing at different times of the day,different days of the week, etc. For example, advertisements appearingat 9:00 a.m. or 7:00 p.m. may receive a greater click through rate thanadvertisements appearing at 4:30 a.m. or 11:00 p.m. Advertisementsappearing during different time periods may be normalized in order tocompare the effectiveness of one or more advertisements. According toyet another embodiment of the invention, a normalized click through rateis calculated using multiple factors. For example, normalization may beperformed for click through rates associated with a certain time of dayin conjunction with click through rates associated with certain usercharacteristics as specified in a user profile.

The scoring component 106 is further operative to calculate a clickthrough rate confidence score corresponding to the normalized clickthrough rate associated with the one or more advertisements comprisingthe result set. The click through rate confidence score corresponding tothe normalized click through rate associated with the one or moreadvertisements comprising the result set comprises an indication of thereliability of the click through data. For example, the normalized clickthrough rate calculated by the scoring component 106 may be discounteddue to one or more factors. According to one embodiment of theinvention, a click through rate confidence score is based upon thequantity of data maintained by the analytics data store. For example, ifthe quantity of data gathered by the analytics data store for a givenadvertisement 108 is below a given threshold, the click through data maynot provide a reliable indication of the frequency with which anadvertisement is selected, thereby resulting in a decreased clickthrough rate confidence score. Alternatively, if the quantity of datagathered by the analytics data store exceeds a given threshold, thenormalized click through rate may provide a strong indication of thelikelihood of another user selecting a given advertisement, therebyresulting in an increased click through rate confidence score.

According to another embodiment of the invention, the click through rateconfidence score is based upon a determination of whether a givenadvertisement has been updated or replaced with an alternateadvertisement. For example, an advertisement may be displayed to usersfor a period of six months. During the time the advertisement isdisplayed, data indicating user interactions with the advertisement ismaintained at the analytics data store 108. If the advertisement isreplaced or updated, the accuracy of the historical click through datamust be discounted, as it does not directly correspond to the newadvertisement. According to one embodiment of the invention, the clickthrough rate confidence score is based upon a comparison between an oldadvertisement and a new advertisement. For example, if the differencebetween an old advertisement and a new advertisement is minimal, theclick through rate confidence score associated with the click throughrate may be increased. Alternatively, if the difference between an oldadvertisement and a new advertisement is significant, the click throughrate confidence score associated with the normalized click through ratemay be decreased. The difference between an old advertisement and a newadvertisement may be measured using a clickability score, which iscalculated according to the methods described herein.

The relevance score, relevance confidence score, normalized clickthrough rate and the click through rate confidence score are combined tocalculate a clickability score, which may comprise a prediction of theclick through rate of a given advertisement in response to a givenquery. According to one embodiment of the invention, the scoringcomponent is operative to use the appropriate combination of relevancescores and click through rates as determined by the amount orsignificance of click through data available. For example, if thequantity of click through data associated with a given advertisement isminimal, the scoring component may give greater weight to the relevancescore and relevance confidence score. Alternatively, if the quantity ofclick through data associated with a given advertisement is significant,the scoring component may give greater weight to the click through rateand click through rate confidence score.

The clickability score is used in conjunction with the bid priceassociated with a given advertisement in response to a given query.According to one embodiment of the invention, the one or moreadvertisements comprising the result set have associated price per clickbids that may be stored in the advertisement data store 104. The bidsassociated with the one or more advertisements comprise the amount ofmoney an advertiser is willing to pay for each user that selects one ofthe advertiser's advertisements. The scoring component 106 is operativeto combine the clickability score and the price per click bidsassociated with the one or more advertisements to compute the expectedrevenue for a given advertisement in the result set.

The expected revenue associated with the one or more advertisementscomprising the result set may be utilized by the ranking/positioningcomponent 110 to order the one or more advertisements. Theranking/positioning component 110 is operative to receive one or moreadvertisements and order the advertisements based upon the one or moreadvertisements' associated expected revenue scores. According to oneembodiment of the invention, the ranking/positioning component 110orders the one or more advertisements by expected revenue in descendingorder.

The ranking/positioning component 110 may be further operative tocalculate the price to charge a given advertiser based upon the positionof an advertisement in a ranked list of advertisements and a givenadvertiser's price per click bid. As previously noted, a price per clickbid may be associated with a given advertisement comprising the resultset. The ranking/positioning component 110 traverses the ordered resultset and determines the minimum price per click that a given advertisermust pay in order to maintain the ranking of a given advertisement inthe ordered result set. The price determined by the ranking/positioningcomponent 110 may be stored in the advertisement data store 104 with anassociated advertisement. The ranking/positioning component 110 deliversthe ordered advertisements to the search engine 112, which may displaythe ordered advertisements to client devices 124 a, 124 b and 124 c thatinitiated the search query. Users of client devices 124 a, 124 b and 124may select one or more of the advertisements displayed. Userinteractions with the displayed advertisements are delivered to theanalytics data store 108.

The system of the present invention may also be used to determine thepositioning of one or more advertisements in a given web page. Users ofclient devices 124 a, 124 b and 124 c communicatively coupled to anetwork 116 may access various web pages. For example, a user may submita search query to a search engine and receive one or more links to webpages responsive to the user's search query. When a user selects a link,the user may be redirected to a web page. The web page displayed to auser may be comprised of both content responsive to the user's searchquery as well as one or more advertisements.

The position of a given advertisement in a web page may affect thelikelihood that a given user viewing the web page will select theadvertisement. For example, a web page displayed to a user may containseveral items of content and advertisements that require a user tonavigate to the bottom of the web page in order to view all the contentand advertisements comprising the web page. A user who fails to navigateto the bottom of the web page may never see, and thereby never select,advertisements displayed on the bottom of the web page. Therefore, asearch engine may want to display the advertisements that generate thegreatest revenue and that are the most likely to be selected in the oneor more areas of a web page that are most frequently seen by users ofclient devices viewing a web page. The system of the present inventionmay further be used to determine the positioning of one or moreadvertisements retrieved from the advertisement data store 104. Theranking/positioning component 110 is operative to utilize the expectedrevenue scores associated with the one or more advertisements comprisingthe result set and determine a position for the one or moreadvertisements on a given web page.

FIG. 2 presents a flow diagram illustrating one embodiment of a methodfor ordering advertisements based upon expected revenue. A search enginereceives a request from a client device, step 200. According to oneembodiment of the invention, a request comprises a search query made upof one or more terms. The search engine uses the request to retrieve oneor more advertisements stored in an advertisement data store, step 201.Advertisements retrieved in response to the request received from aclient device may be retrieved according to one or more matchingmethods. According to one embodiment of the invention, advertisementsconstituting an exact match are retrieved from the advertisement datastore. An exact match may comprise an advertisement that contains all ofthe terms of a given query. According to another embodiment of theinvention, advertisements constituting an advanced match are retrievedfrom the advertisement data store. An advanced match may comprise anadvertisement wherein the advertisement's title, category information,and topic all correspond to the query received. Those of skill in theart recognize additional methods for matching and retrievingadvertisements in response to a given search query may be utilized.

A result set is generated comprising the one or more advertisementsretrieved from the advertisement data store, step 201. A relevance scoreis calculated for the one or more advertisements comprising the resultset, step 202. According to one embodiment of the invention, a relevancescore comprises an indication of how relevant a given advertisement isto the one or more terms comprising a given search query. For example,an advertisement retrieved from the analytics data store may comprisecontent as well as one or more items identifying the advertisement, suchas title, topic, category, etc. A relevance score may calculate howclosely a given advertisement's title, topic, category, etc., as well ascontent, relate to a given query.

A relevance confidence score is calculated corresponding to therelevance score for the one or more advertisements comprising the resultset. The relevance confidence score may provide an indication of theextent to which a given advertisement matches a given query as well asthe reliability of the given advertisement. According to one embodimentof the invention, relevance confidence scores are based upon theadvertiser providing a given advertisement. For example, if anadvertisement was received from a reputable source, the relevanceconfidence score associated with the given advertisement may beincreased. Alternatively, if an advertisement was received from anunknown business, the relevance confidence score associated with theadvertisement may be decreased. According to another embodiment of theinvention, relevance confidence scores may be based upon the type ofmatch that resulted in a given advertisement's inclusion in the resultset. For example, the relevance confidence score associated with anadvertisement constituting an exact match may be increased, whereas therelevance score associated with an advertisement constituting anadvanced match may be decreased. According to another embodiment of theinvention, relevance confidence scores may be assigned or modified basedupon an examination of an index identifying unreliable sources. Forexample, an index may be maintained identifying advertisement URLs thatdirect users to questionable or unreliable content, thereby resulting ina decreased relevance confidence score. Relevance confidence scores areassigned to the one or more relevance scores associated with the one ormore advertisements comprising the result set.

Analytics data is retrieved to calculate a normalized click throughrate, step 204. According to one embodiment of the invention, ananalytics data store maintains data identifying user interactions with agiven advertisement. For example, the analytics data store may maintaindata identifying how many times a given advertisement was selected, thetype of users that selected a given advertisement, the time of day anadvertisement was selected, the position of an advertisement in a rankedlist when selected, how often an advertisement was displayed, etc. Theanalytics data retrieved from the analytics data store is used tocalculate a normalized click through rate, step 208. A normalized clickthrough rate allows for an effective comparison of one or moreadvertisements displayed at different times of the day, differentpositions in a ranked list, etc. According to one embodiment of theinvention, normalized click through rates are calculated by determiningthe ratio of the number of times a given advertisement was selected at agiven position in a ranked list and the number of times it was expectedthat a given advertisement would be selected when displayed in a givenposition in ranked list. Reference data may be used to provideinformation indicating the expected number of selections a givenadvertisement should receive when displayed in a given position.

Table A presents an equation illustrating one embodiment for computingthe expected click-through rate for a given advertisement displayed in agiven position in a ranked list of advertisements: TABLE A${{expected}\quad{clicks}} = {\sum\limits_{r = 1}^{n}\quad{{ctr}_{r}*{imps}_{r}}}$In the equation presented in Table A, ctr_(r) is the reference clickthrough rate for a given rank r and imps_(r) is the number of times agiven advertisement was displayed at a given rank. According to oneembodiment of the invention, the number of times an advertisement wasselected is divided by the expected number of selections for a givenadvertisement to compute the normalized click through rate. A normalizedclick through rate is calculated for the one or more advertisementscomprising the result set, step 206.

The result set is traversed and a check is performed to determinewhether an advertiser has changed or modified any of the one or moreadvertisements comprising the result set, step 208. If it is determinedthat an advertisement has been changed or modified, the extent of changebetween the old advertisement and new advertisement is calculated, step209. According to one embodiment of the invention, a clickability score(calculated according to methods described herein) may be associatedwith each advertisement stored in the advertisement data store and mayprovide a prediction of the probability that a given user will select agiven advertisement. A clickability score may be used to compare thechange between an old advertisement and a new advertisement. Accordingto one embodiment of the invention, a clickability score is a value fromzero to one.

A date associated with each version of a modified advertisement isretrieved, step 210. The date associated with a modified advertisementallows the scoring component to associate click through data with theproper version of an advertisement. For example, if an initial versionof an advertisement has an associated date of “Jun. 1, 2004” and anupdated version of the advertisement has an associated date of “Dec. 1,2004,” the scoring component may determine that click through data forthe period of Jun. 1, 2004 to Nov. 30, 2004 corresponds to the initialversion of the advertisement. The scoring component may furtherdetermine that click through data from Dec. 1, 2004 onward is associatedwith the updated version of the given advertisement. Thus, the dateassociated with a given advertisement allows the scoring component toascertain the version of an advertisement associated with a given periodof click through data retrieved from the analytics data store.

The normalized click through data associated with a given advertisementis used to calculate a click through rate confidence score, step 212.According to one embodiment of the invention, a click through rateconfidence score is based upon the quantity of data gathered by theanalytics data store. For example, if the quantity of data gathered bythe analytics data store for a given advertisement is minimal, the clickthrough data may not provide a reliable indication of the frequency withwhich an advertisement is selected, thereby resulting in a decreasedclick through rate confidence score. Alternatively, if the quantity ofdata gathered by the analytics data store is significant, the normalizedclick through rate may provide a strong indication of the likelihoodthat a given user will select a given advertisement.

If an advertisement has been modified, the date and difference inclickability score are used to calculate a click through rate confidencescore associated with the normalized click through rate for a givenadvertisement, step 212. For example, if a given advertisement has beenchanged significantly, click through data corresponding to the oldversion of the advertisement may fail to provide an adequate indicationof the click through rate for the new version of the advertisement.Thus, the click through data corresponding to the old version of theadvertisement may be given less weight when calculating the clickthrough rate confidence score, while click through data for the newversion will be given greater weight in calculating the click throughrate confidence score.

In addition to the foregoing, if a given advertisement has been changedminimally, click through data corresponding to the old version of theadvertisement may provide an indication of the click through rate forthe new version of the advertisement, as the small change may not affectperformance of the advertisement significantly. Thus, if the differencein clickability score is a value near one, thereby indicating that thedifference between an old advertisement and new advertisement issignificant, less weight may be given to the click through dataassociated with the old advertisement. Alternatively, if the differencein clickability score is a value near zero, thereby indicating that thedifference between an old advertisement and a new advertisement isminimal, greater weight may be given to the normalized click throughdata for the old version of the advertisement when calculating a clickthrough rate confidence score.

The relevance score, relevance confidence score, click through rate andclick through rate confidence score are combined to calculate aclickability score for the one or more advertisements comprising theresult set, 214. A clickability score provides an indication of thelikelihood that a given user will select a given advertisement inresponse to a given query. According to one embodiment of the invention,the clickability score is calculated using the appropriate combinationof relevance scores and click through rates as determined by the amountor significance of click through data available. For example, if thequantity of click through data associated with a given advertisement isminimal, greater weight may be given to the relevance score andrelevance confidence score. Alternatively, if the quantity of clickthrough data associated with a given advertisement is significant,greater weight may be given to the click through rate and click throughrate confidence score.

The clickablity score is used to calculate the expected bid revenue ofeach advertisement comprising the result set, step 215. The expected bidrevenue of a given advertisement provides an indication of the revenue asearch engine may receive from a given advertiser if a givenadvertisement is displayed in a given position of a ranked list ofadvertisements. Associated with the one or more advertisements retrievedfrom the advertisement data store is a value indicating an advertiser'sbid for displaying an advertisement in a given position in a ranked listof advertisements. For example, an advertiser may provide a bid fordisplaying a given advertisement in a given position in a ranked list ofadvertisements in response to a given query. In a bidding marketplacefor advertisements, a bid associated with a given advertisementindicates the maximum amount of money an advertiser is willing to pay todisplay an advertisement in a given position in a ranked list ofadvertisements in response to a given query.

The product of the clickability score for a given advertisement and thebid associated for displaying a given advertisement in a given positionin response to a given query provide the expected bid revenue of a givenadvertisement. The one or more advertisements comprising the result setmay be ordered by each advertisement's associated expected bid revenuevalue, step 216. Table B illustrates one embodiment of an equation forcomputing the expected bid revenue of one or more advertisementscomprising a result set: TABLE B${Rev} = {\sum\limits_{k = 1}^{P}\quad{{{CLKB}\left( {Q,L_{k}} \right)} \cdot {{bid}\left( {Q,L_{k}} \right)}}}$In the equation presented in Table B, CLKB(Q,L_(k)) is the clickabilityof (Q,L_(k)), where Q is a given query, L_(k) is an advertisement servedin position k of a ranked list of advertisements in response to query Q,bid(Q,L_(k)) is the bid price a given advertiser is willing to pay tohave an advertisement L displayed in position k in response to query Q,and P is the number of advertisements comprising a result set.

Though a bid provides a maximum value a given advertiser is willing topay, in a bidding marketplace a bid may not necessarily be the amount ofmoney an advertiser actually pays for having an advertisement displayedin a given position. For example, in a bidding marketplace whereadvertisers are unaware of other advertisers' bids, an advertiser's bidmay represent the maximum value an advertiser is willing to pay for agiven advertisement to be displayed in a given position. A firstadvertiser may bid $1.25 to have an advertisement displayed in responseto a given query, and a second advertiser may bid $1.50 to have anadvertisement displayed in response to the same query. The actual pricecharged to the second advertiser may be $1.26, as this represents theminimum bid necessary for the second advertiser to exceed the firstadvertiser's bid and maintain a given position in a ranked listing ofadvertisements. Because an advertiser may only be charged the priceneeded to exceed a lower bid, the actual amount of money a givenadvertiser pays is needed in order to calculate the expected revenue ofa given advertisement.

To compute the expected revenue of a given advertisement, a bubblepopping algorithm may be used to determine the actual price a givenadvertiser will pay to maintain a given position for a givenadvertisement displayed in a ranked list of advertisements, step 218.The result set, ordered by expected bid revenue, is delivered to clientdevices, step 220. The result set is displayed as a ranked list ofadvertisements. A user of a client device may select one or more of thegiven advertisements displayed to the user.

As described above, a bubble popping algorithm may be used to calculatethe expected revenue of one or more advertisements comprising a resultset. FIG. 3 presents a flow diagram illustrating one embodiment of amethod for using a bubble popping algorithm to calculate the actualprice paid by a given advertiser in a bidding marketplace when anadvertiser's advertisement is displayed in a given position in a rankedlist of advertisements in response to a given query.

A first advertisement is selected from a result set ordered by expectedbid revenue, step 302. Additionally, the result set is traversed toselect a second advertisement from the result set, step 304. Accordingto one embodiment of the invention, the second advertisement selectedcomprises an advertisement ranked subsequent to the first advertisementselected. The expected bid revenue associated with the secondadvertisement, which may be calculated in accordance with the equationillustrated in Table B, is retrieved, step 308. The expected bid revenueis used to calculate the actual price charged for the firstadvertisement in order for the first advertisement to maintain itsposition in the result set ordered by expected bid revenue, step 310.The expected bid revenue of a given advertisement may comprise theproduct of a given advertisement's clickability score and associatedbid. The bid associated with a given advertisement is decreased to theminimal value needed in order for the advertisement's expected revenueto exceed a subsequently ranked advertisement's expected bid revenue. Ifa second advertisement is not found that is ranked consecutively afterthe advertisement selected, the selected advertisement comprises thelast advertisement in the result set. The last advertisement is assignedan actual price value that results in the last advertisement having anexpected revenue value below all other expected revenue valuesassociated with the one or more items comprising the result set, step306. Table C presents an equation illustrating one embodiment forcomputing the actual price of a given advertisement based upon theexpected bid revenue of a subsequent advertisement in a result setordered by expected bid revenue. TABLE C${{paid}\left( {Q,L_{k}} \right)} = {\frac{{CLKB}\left( {Q,L_{k + 1}} \right)}{{CLKB}\left( {Q,L_{k}} \right)} - {{bid}\left( {Q,L_{k + 1}} \right)} + {{\$ 0}{.01}}}$

In the equation presented in Table C, CLKB(Q,L_(k)) is the clickabilityof (Q,L_(k)), where Q is a given query, L_(k) is an advertisement servedin position k of a ranked list of advertisements in response to query Q,bid(Q,L_(k)) is the bid price a given advertiser is willing to pay tohave an advertisement L displayed in position k in response to query Q,paid(Q,L_(k)) is the actual price paid by a given advertiser when anadvertisement L is displayed in position k in response to query Q.

The actual price computed for the one or more advertisements comprisingthe result set may be used to calculate the expected revenue of the oneor more advertisements. Table D presents an equation demonstrating oneembodiment for computing the expected revenue of one or moreadvertisements comprising a result set: TABLE D${Rev} = {\sum\limits_{k = 1}^{P}\quad{{{CLKB}\left( {Q,L_{k}} \right)} \cdot {{paid}\left( {Q,L_{k}} \right)}}}$In the equation presented in Table D, CLKB(Q,L_(k)) is the clickabilityof (Q,L_(k)), where Q is a given query, L_(k) is an advertisement servedin position k of a ranked list of advertisements in response to query Q,paid(Q,L_(k)) is the actual price paid by a given advertiser when anadvertisement L displayed in position k in response to query Q, and P isthe number of advertisements comprising a result set.

As discussed above, embodiments of the present invention may determinethe positioning of one or more advertisement in a given web page. FIG. 4presents a flow diagram illustrating one embodiment of a method fordetermining the location of one or more advertisements in a given webpage displayed to a user of a client device. A result set is retrievedcomprising one or more advertisements ordered by expected revenue, step402. A web page is divided into one or more locations at which one ormore advertisements from the result set may be displayed, step 403.According to one embodiment of the invention, a web page is divided intoa North position, South position and East position. A check is performedto determine whether any space is available in the North position of agiven web page for displaying a given advertisement from the result set,step 404. If space is available in the North position of the given webpage, the ordered result set is traversed and an advertisement isselected, step 406. The click through rate confidence score associatedwith a given advertisement's click through rate is retrieved, whichprovides an indication of the likelihood that a user will select thegiven advertisement. If the click through rate confidence scoreassociated with the advertisement selected exceeds a predeterminedconfidence threshold, step 408, the advertisement is placed in the Northposition of the given web page, step 412. If there is no available spacein the North position, step 404, or if a given advertisement's clickthrough rate confidence score does not exceed the confidence threshold,step 408, the advertisement selected, and the one or more advertisementscomprising the result set ranked below the selected advertisement, areplaced in the East position of the web page, step 410.

An advertisement is selected from the one or more advertisements placedin the North position, step 414. The selected advertisement's associatedexpected revenue is compared with an expected revenue threshold, step416. If the expected revenue threshold is greater than anadvertisement's associated expected revenue, the advertisement selected,and the one or more advertisements placed in the North position rankedbelow the given advertisement, are moved to the East position, step 424.The top N ranked advertisements in the East position are replicated andmoved to the South position, step 428. Alternatively, if theadvertisement's expected revenue exceeds the expected revenue threshold,step 416, a check is performed to determine whether all advertisementsplaced in the North position have been examined, step 420. If all of theone or more advertisements placed in the North position have beenexamined, processing terminates, step 428. If all advertisements havenot been examined, step 420, a next advertisement is selected, step 414.The resulting web page displays advertisements that generate thegreatest revenue and have the greatest likelihood of being selected inlocations where they are most likely to be viewed and selected. Those ofskill in the art recognize that the method illustrated in FIG. 4 may beused to place advertisements in any of the one or more locations of agiven web page.

While the invention has been described and illustrated in connectionwith preferred embodiments, many variations and modifications as will beevident to those skilled in the art may be made without departing fromthe spirit and scope of the invention, and the invention is thus not tobe limited to the precise details of methodology or construction setforth above as such variations and modifications are intended to beincluded within the scope of the invention.

1. A method for determining the performance of a plurality of versionsof a given advertisement, the method comprising: retrieving a firstversion of an advertisement and associated click through data;retrieving a second version of the advertisement and associated clickthrough data; calculating a clickability score for the first version ofthe advertisement using the click through data associated with the firstversion of the advertisement; calculating a clickability score for thesecond version of the advertisement using the click through dataassociated with the second version of the advertisement; determining adifference in clickabilty scores between the first advertisement and thesecond advertisement; and modifying the clickability score associatedwith the first version of the advertisement based upon the difference inclickability scores.